Behind the Scenes: Linux Install Fest Lab at Engage 2026

Bill, Matijn and I did three hands on sessions at Engage.
If you wondered how we have been doing it. Here is how this worked and it could be leveraged for own workshops.
Using cloud hosted servers can make a lot of sense. And it is very cost efficient.

Hetzner is a provider with a very modern and easy to use GUI. But they also have a great API to create and manage servers.
I am using them for years for DNUG and other events. And I would make it available for friends & family.

Recently Hetzner moved their separate DNS API into the central cloud API -->https://docs.hetzner.cloud/reference/cloud
I took this opportunity to implement most of this API in an existing Lotus Script class (Script Lib) and read the full inventory for a project to manage it from a Notes database.
Each database would match 1:1 a Hetzner project. We created a private network for the project and offered services like NFS based file mounts to provide workshop data.

The database is mainly designed for lab setups with the same server configuration used for all servers.
A profile document is used to define the type, size, location etc of the server.
What I also added is support for cloud.init configurations.

For our servers I just needed a host name. The e-mail address is used for an integrated mail workflow to notify users -- including sending them their server name and IP.
When creating a new server also a DNS record is created and also the IN-ARPA entry is set properly.

Creating a new server just takes a couple of seconds. In our case you have to wait a moment because we installed additional software, configured a NFS mount and rebooted the machine.

Because we did not expect everyone to register ahead of time thru the official Engage registration processing a mail flow from the Leap App to our database, I added a registration form to feed in new server requests directly.
The slides had a registration QR code to fill in the registration code.

This gave us a very flexible way to generate new servers on-the-fly.

The whole lab setup was done in a reusable way. So all the time spent on it wasn't just for this one conference.
And I will extend it over time for other use cases. The Lotus Script Lib is the core of the application, but with my additions also reading the inventory could be re-used in other areas.

This is a good example use case for a Notes database. It would probably have taking me much longer with most other approaches.

See some details below. I also added label support and to specify firewalls directly or via labels.
The could.init replaced my earlier Ansible scripting I used to customize servers after they are created.
But I also added a Ansible host list export and we used the same SSH key for all servers beside the management server which hosted the file shares.

-- Daniel

         

• Comments [0]

Domino On Proxmox LXC Containers Part II

As mentioned at Engage, I am working on Domino container deployments on Proxmox.
The first deliverable was a new build option for the container image to create two components:

• A LXC template which can be cloned into a new LXC container using pct command line
• A /opt volume which is read-only mounted into each container

You can see from the configuration dump below how this would look like.
The second deliverable is a management script "dompct" which allows to create and manage LXC containers.
I added some screen shots below.

The new script is already part of the develop branch of the Domino Start script, but I am still enhancing and testing it.
It comes with configuration profiles which can be managed and assigned to servers when created.

This includes ZFS volumes assigned to LXC containers -- as you can also see from the example below.
The new option build.sh -pct and the new dompct tool standardize and simplify Proxmox LXC deployments.

There isn't full documentation yet in the GitHub project. But dompct has command line help already.
You can either use command-line parameters or the menu.
The menu prompts to select a container if not specified. Or a profile if not specified.
It also prompts for the hostname when creating a new container and the hostname is not specified at command-line.

The tool can be used interactively or for automated provisioning. That's why I also added -json for output.

The Proxmox LXC is a first implementation and I mainly wrote it for one customer who is actively looking into Domino on Proxmox deployments.
But I wanted to make it available already for feedback.


--- Configuration output ---

--------------------------------------------------------------------------------
Config LXC 800
--------------------------------------------------------------------------------

arch: amd64
cores: 4
features: nesting=1
hostname: domino-01
memory: 8192
mp0: /rpool/data/domino-opt-20260419-1053,mp=/opt,ro=1
mp1: /rpool/data/subvol-800-domino-local,mp=/local
mp2: /rpool/data/subvol-800-domino-nsf,mp=/local/notesdata
mp3: /rpool/data/subvol-800-domino-translog,mp=/local/translog
mp4: /rpool/data/subvol-800-domino-daos,mp=/local/daos
mp5: /rpool/data/subvol-800-domino-backup,mp=/local/backup
net0: name=eth0,bridge=vmbr0,hwaddr¼:24:11:75:9A:30,ip=dhcp,type=veth
ostype: ubuntu
rootfs: local-zfs:subvol-800-disk-0,size=20G
swap: 0
tags: domino
unprivileged: 1

description:
HCL Domino server 800









--- Configuration profile example ---


# Domino LXC configuration - default

PCT_TAGS=domino
PCT_DATA_POOL=rpool/data
PCT_DAOS_POOL=rpool/data
PCT_TRANSLOG_POOL=rpool/data

PCT_TRANSLOG_SIZE_GB=5
PCT_NSF_SIZE_GB=100
PCT_BACKUP_SIZE_GB=100
PCT_DAOS_SIZE_GB=100

PCT_RAM_GB=8
PCT_SWAP_GB=0
PCT_CPU=4

env_DOMSETUP_ENABLED=1
env_NODE_EXPORTER_OPTIONS=default


PCT_NET0_TEMPLATE=name=eth0,bridge=vmbr0,ip=%PCT_IP%/24,gw=192.168.96.98



--- Command Line Parameters ---


dompct - LXC Container Control Utility

Usage:

./dompct.sh <command> [VMID] [options]

Commands:

create              Create new container
start               Start container
stop                Stop container
status              Show container status
enter | bash        Enter container shell
config              Show container configuration
update              Update container
destroy             Destroy container
about               Show container information
profile             Select or apply profile
list                List containers
KILL                Force kill container
KILL-WITH-DISKS     Kill container and remove disks

Global Options:

-profile=<name>     Use profile (from ~/.dompct/*.cfg)
-host=<name>        Set hostname
-hostname=<name>    Same as -host
-tags=<tags>        Set Proxmox tags (comma-separated)
-ip=<ip>            Assign IP address
-description=<txt>  Set container description
-opt-vol=<opts>     Volume options (advanced)

Output Options:

-json               Output in JSON format (where supported)

Arguments:

VMID                Numeric container ID (required for most commands)

Examples:

./dompct.sh list
./dompct.sh start 800
./dompct.sh create -profile=mail
./dompct.sh profile
./dompct.sh destroy 800

Notes:

- If VMID is omitted, interactive menu may be used
- Profiles are stored in: ~/.dompct/

• Comments [0]

Ubuntu 26.04 LTS released with a last minute surprise -- Kernel 7.0

Today finally the release shipped and they just updated the website and also provided all ISO images.
I have been using the release candidate for a while natively and for Docker containers.

• Desktop
• Server
• WSL
• Container Image default for ubuntu is also version 26.04.

--- Kernel ---


The RC kernel was: 6.19.0-9-generic
The released kernel : 7.0.0-14-generic

--- GLIBC ---

GLIBC is also very up to date: (Ubuntu GLIBC 2.43-2ubuntu2) 2.43


The main challenge here is a major new kernel release.
The change isn't as big as it sounds on top of the latest 6.19.
Ubuntu might have waiting for the kernel to be released and updated it because of newer CPU support in kernel 7.0 which they wanted to be prepared for.

But this raises questions for Domino on Ubuntu 26.04 LTS.
I would not update any production environments yet.
This is way too new. But it works well in my personal testing.

If you give it a try in a test environment I would be interested in your findings.
But given the major kernel update, I would really wait for any production deployment.


Official release statement for 14.5.1

https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0128491

Equivalent OS with the following kernel/packages:

kernel-6.12.0-55.9.1.el10_0.x86_64 or higher 6.12 kernel
glibc-2.39-37.el10.x86_64 or higher
libstdc++-14.2.1-7.el10.x86_64 or higher

You can see that from Kernel level and also GLIBC level Ubuntu 26.04 is quite more current then what is listed as officially tested and supported for Domino.

Note about Ubuntu system requirements

There has been a change in system requirements for the deskop. The hardware resource requirements have been bumped up a bit.
On the server side the hardware resource requirements did not change. 1,5 GB is still a pretty low requirement.
With Domino I would at least have 4 GB for a small server anyhow. But the 1,5 GB show the low resource requirements a Linux server has -- on top of the application requirements.

• Comments [0]

Engage 2026 Presentation Slides Leveraging CertMgr and Resources

As promised at Engage this week, here is my presentation including additional material as promised in the session.
The Script Lib is a first version which I wrote for one of my projects to integrate certificate management.
I am happy to share it and I am looking forward to feedback.

What I also added is the code I used in my demo to integrate with HashiCorp.
For the presentation I also setup a new HashiCorp development project which might help to get started with HashiCorp.

The HashiCorp instance I was using in my demo has been automatically setup using this project.
This includes the provisioners for ACME and API level integration.

I hope the resources help you to get started with your own integrations.

HashiCorp is an interesting option for certifcate management inside a company.
This includes ACME support and also the secrets engine to securely distribute secrets like TLS private keys.

-- Daniel


Engage Presentation slides

https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/blob/main/presentations/engage_2026_certmgr.pdf

Example Integration: CertMgr HashiCorp

https://opensource.hcltechsw.com/domino-cert-manager/hashicorp/

Script Lib Source code

https://github.com/HCL-TECH-SOFTWARE/domino-cert-manager/tree/main/resources

HashiCorp Deployment Project to get started with HashiCorp

https://github.com/nashcom/nsh-vault-deploy

• Comments [6]

Performance Challenges on modern Intel Hybrid CPUs with Notes and other applications

After moving to a new notebook, my Notes client got very slow when I was in a Sametime meeting or ran other a bit more CPU intensive tasks.
It turns out that Windows is giving the Notes client the slower E-Core CPU cores, which results in a very slow UI responsiveness.

HCL is aware of this and we had a call last week.
If you are running a modern Intel CPU on your workstation/notebook you should read this link for detailed information and a tool

https://github.com/nashcom/nshcpuset

I wrote up all the technical details and a small troubleshooting and work-around tool.

My take away from this is that we have to watch more for the CPU P-Cores then the total number of cores a modern CPU has.

The notebook I got has 2 physical P-Cores with Hyper-Threading + 8 E-Cores.
This doesn't hit Notes alone! We had interesting experiences with VMware workstation before they looked into it.

Darren blogged about the VMware issues he ran into  https://blog.darrenduke.net/darren/ddbz.nsf/dx/type-2-hypervisors-and-the-evils-of-e-cores.htm

I am not a fan of Processor Lasso and looked into the APIs on my own to get a clear picture.
Processor Lasso is a too big tool with a lot of options.
You can also use it, but the easier path is to use my small program.

-- Daniel

• Comments [0]

Domino on Proxmox deployment models

With the current cost increases for VMware licenses and also hardware some of my customers start to look very seriously for alternate solutions.
One deployment method would be Proxmox with local ZFS disks.

ZFS is a very interesting file-system and volume manager combined. It offers a lot of flexibility and choices.
Compression, de-duplication, optimized record size, snapshots, encryption and more benefits.

I am a big fan for a while and I did blog about Domino on Proxmox before.
Last weekend I looked into different deployment methods for Domino on Proxmox for a customer.

A VM with Linux + Docker + Domino might be additional overhead which could be optimized.

LXC containers use native ZFS volumes and are very efficient:

• Lightweight Linux instance with shared kernel like Docker
• Leverage ZFS host subvolumes for a true end to end storage management and less overhead
• Standardized OS images

What is missing is the automated installation we know from Docker.
I am looking into that right now and if you are interested in Domino on Proxmox, I want to hear from you.

At Engage I will show case Domino on Proxmox. I will bring a Proxmox server running on my notebook with an automated Domino installation including management scripts with me.
I would like to hear from you what type of deployments you are looking into.

We are working on multiple deployment options optimized for Proxmox in combination with Domino clustering to optimize the deployment footprint.
This includes DAOS storage deduplication cross servers and backup.

-- Daniel

My Notebook deployment on VMware workstation with Proxmox 4 CPU cores + 8 RAM as a test/demo installation for on the road.
For LXCs it does not even need the hardware support.

• Comments [0]

Join Us at Engage 2026: Domino on Linux, Installfest and CertMgr

Engage Conference 2026 is just around the corner. A quick look at the agenda reveals several Linux-focused sessions—and that’s no coincidence. Linux continues to play a key role in sovereign and future-proof solutions. As in previous years, Bill is hosting the popular Domino on Linux round table. In addition, the conference will feature a Linux desktop session along with a series of Domino on Linux Installfest sessions. These sessions can be attended independently, but they are also designed to complement each other and build progressively. Bill, Martijn and I have teamed up to present a full range of sessions—from beginner through to expert level. This includes hands-on labs using on-demand virtual machines at Hetzner, along with forward and reverse DNS within our domino-lab.net environment. Even if you can’t attend the Installfest sessions, the Linux round table is highly recommended. It’s a great opportunity to ask questions, share feedback, and connect with others—especially if you’re already running Domino on Linux or planning to explore it. For example on Proxmox. We’ll also share updates on what has been developed over the past year, along with a new initiative aimed at making Domino on Linux more accessible for administrators. As part of this effort, a new repository has been launched as a central entry point into the Domino on Linux ecosystem: https://nashcom.github.io/nsh-domino-linux/ Beyond the Linux sessions, I will present another session on Domino CertMgr. If certificate management and automation are topics of interest, this session should definitely be on your list. You can explore the full agenda here: https://engage.ug/pages/session2026 We are looking forward to seeing you at Engage 2026. Bill | Martijn | Daniel

• Comments [0]

Certificate Lifetimes Are Shrinking — Is Your Domino Infrastructure Ready?

Certificate maximum lifetimes dropped to 200 days in March 2026 and will reach 47 days by 2029.
At that frequency, manual renewal becomes operationally impossible. HCL Domino CertMgr automates issuance and renewal end-to-end.
This includes certificate rollover and also key rollover -- which is as important as rolling over certificates and often overlooked in current discussions.

For everything outside Domino — NGINX, load balancers, and other services — there is a need for automated certificate management.
Rotating the private key on every renewal cycle is the part most deployments have not solved yet.

Here is a longer document I wrote up for one of the projects with additional details:

https://github.com/nashcom/srvguard/blob/main/docs/certificate-lifetime-reduction.md

This initiative started last week. The timing is not a coincident. It's in time for my Engage presentation and the latest changes for certificate lifetime.

When HCL introduced CertMgr in Domino 12.0 most of the feature we have today have been already present.
Domino 12.0.1 introduced export / import which might be helpful for automation.

CertMgr and certstore.nsf are built on open standards and importing certificates/keys and handling CSRs for an automated flow are straightforward to implement on Domino CertMgr side.
The challenge is most time the CA side. My previous post shows a straightforward HashiCorp configuration using ACME as the protocol.
But there are also other easy to use ways to integrate with modern CAs.

There is more to come. But I want to keep also some news for my conference session.
If you are curious what is coming you can take a look at the referenced projects.

-- Daniel

• Comments [1]

HashiCorp ACME with Domino CertMgr – a Beautiful Combination

For my upcoming session at HCL Engage next month, I’ve been looking into additional integrations for Domino CertMgr. The guiding principle is simple: use standards wherever possible.
One of the most important standards in this space is ACME. It has become the default protocol for automated certificate lifecycle management and is supported by virtually every modern toolchain.

Vault as an enterprise ACME CA

HashiCorp Vault is a modern, API-first PKI solution widely used in corporate environments. With built-in ACME support, Vault can act as a fully functional ACME certificate authority.
That makes integration straightforward:

• Vault provides the CA
• ACME provides the interface
• CertMgr consumes certificates
  

No custom code, no special handling—just standard protocol.

Why this combination works so well

Domino CertMgr was designed for automation. Pairing it with Vault via ACME creates a clean and robust setup:

• enterprise-grade CA
• fully automated issuance and renewal
• standard-based integration

Current work

I’m currently building a streamlined Vault setup to make testing and demos easier, including ACME-enabled configurations out of the box.
This allows quick validation of:

• Domino integrations
• short-lived certificates
• policy-driven issuance

Some of this will be shown at Engage conference.

• Comments [0]

Explaining the Domino CScan token

Now that ClamAV integration shipped in 14.5.1 hopefully more admins look into CScan.
CScan for mail flow scan is very straightforward to configure. The configuration database document now defaults to ClamAV with the default parameters.

There some details about the implementation which are not well known but eventually good to know.
There is a Scan Token added when the document is scanned. This token avoids rescan the document on the next hup as long the virus scan signature does not change.

Technically the token is a JWT which you can decode and read. Each server creates a key stored in it's CScan server document.
The key is encrypted for the server and there is a public key to validate the token.

The token contains the information about the server and time when the document was scanned, a hash, a thumb print of the signing key to find the right signing key.
It contains also the scan version and pattern and the configuration.
There is also a hash built based on the attachments in some way to avoid re-scanning and to check if attachments changed.

Here is an example token which could be useful to know:

Field Name: $$CScanToken
Data Type: Text
Data Length: 644 bytes
Seq Num: 1
Dup Item ID: 0
Field Flags: SUMMARY

"eyJ0eXAiOiAiSldUIiwgImFsZyI6ICJFZERTQSJ9.eyJ2ZXJzaW9uIjoxLCJpc3N1ZXIiOiJEb21pbm8gQ29udGVudCBTY2FuIiwiY3JlYXRlZCI6IjIwMjYwMzIxVDE5NDE1OSwxMyswMCIsInNlcnZlciI6IkNOPXJheS5sYWIuZG51Zy5ldS9PPWRudWctbGFiIiwic2NhblZlcnNpb24iOiJDbGFtQVYgMS41LjEvMjc5NDciLCJjb25maWdEYiI6IjAwMjU4ODUyMTA3RUQ1NTIiLCJjb25maWdJRCI6IkIyODdDRTBDOUM1N0NCMkUwMDI1OEFCMTAwNTQxNjg1IiwiY29uZmlnTmFtZSI6ImNsYW1hdi1sYWIiLCJ2ZXJpZmljYXRpb25IYXNoIjoiRjJDM0IwOTA1RTAxMjQ0Qzk3Qjg5MDJDNzI3MjVEOUQ4RENDMERGMSIsImtleVRodW1icHJpbnQiOiJzMTFsSUxDeWVfbUk3NEpGZmlkYm5wbWsxY1EiLCJoYXNoQWxnb3JpdGhtIjoiU0hBMSJ9.Pe8ntQ0WaiDH8xksK2gK_8034uRul4qkFWD5GYp1iZKBET1_D-vqsiFs35X0DqUgNCHACWD0wINJ3ErE5OqbAw"

---


{
 "version": 1,
 "issuer": "Domino Content Scan",
 "created": "20260321T194159,13+00",
 "server": "CN=ray.lab.dnug.eu/O=dnug-lab",
 "scanVersion": "ClamAV 1.5.1/27947",
 "configDb": "00258852107ED552",
 "configID": "B287CE0C9C57CB2E00258AB100541685",
 "configName": "clamav-lab",
 "verificationHash": "F2C3B0905E01244C97B8902C72725D9D8DCC0DF1",
 "keyThumbprint": "s11lILCye_mI74JFfidbnpmk1cQ",
 "hashAlgorithm": "SHA1"
}

• Comments [0]